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1. Atetract 


<jt ■ci"" lint ^an irchltactur# for lapXtMntlno th# proctta-Xtval decision Mklng for 
m bitrtrddcaXXy atructurad taltrobot currently balng iapXmntad at tht Jet Propulsion j 
Laboratory t JPL) ^Constraints on tha architecture dttign, architect ure partitioning concepts. j 
detailed description of th* f existing and proposed lapleaentatlona are provided. 
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2. Introduction 
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The architecture of a telerobot le required to support autonoaoue and teleoperated 
actlvitiee in a Banner designed to obtain the aaxlaua synergy of function between robotlce 
and teleoperation. The JPL telerobot lepleaente auch a ayatea for applications involving 
various servicing and repair operations In apace. 

The architecture propoeed for the JPL telerobot decoapoeee the functionality of syatea 
operations into three hierarchical levels. For the autonoaoue component of the architecture, 
theee level* conaiet of the Task Planning Level at the top of the hierarchy, followed by the 
Process- Level at the lntereedlate level, and the Actuation _and Sensor Level at the bottoi of 
■the hierarchy. Each of these levels is designed to operate robustly by using local feedback 
to detect and recover froe local error*. 


The Task Planning level Is concerned with the overall context of the taek. including global 
planning, overall execution eonitorlng and task replanning. The basis for the decision making 
Is the Task Sequence Logic vlth relatively little Incorporation of Task Doaaln Physics. As a 
consequence, activity tlelng at this level la driven by taek scheduling requlreaents. and the 
decision making methods used are predominantly symbolic and not numeric. In the JPL telerobot 
the functions at this level are mechanized by an Artificial Intelligence Planner subsystem 
<AXP> vlth the logical planning of a Module swap-out sequence being a representative example 
of internal subsystem activity. 

The Procees Level la concerned vlth the planning, execution and monitoring of subtasks such 
as grasping objects, object seeeably etc. Actlvitiee at this level require only a local 
subtaak context vithin which various eeneor and actuation subsystems are commanded and 
coordinated to accomplish a given eubtaek. Decialon aaking at this level hae to be cognizant 
of the phyeice and geometry of the taek doaaln and the occurrence of various run-tise 
physical events In the telerobot and lta environment. Decision aaking la characterized by a 
hybrid of aymbollc and numeric processing with activity timing determined by the event line 
of physical sensed events. Mechanization at this level la provided by a Run-Tiee Control 
subsystem CRTC) vhlch performs the various trajectory determinations, reflex specif lea t lone 
etc. for subtasks such as grasp centering and activation, parts sating etc. 

The Actuation and Sensor Level serves to provide the basic manipulation and sensor 
capabilities of the robot. The activity at this level is Intimately tied to the phyeice of 
the domain but no task context is required. Activity occurs in continuous tlae (practically 
realized by high rate servo sampling) and is predominantly numeric in nature. Functions at 
this level are ■echanized by the Manipulation and Control Mechanization aubeyetee (MCH) and 
the Sensing and Perception subeystea (SAP) with compliant motion execution or object tracking 
constituting representative examples of subsystem activity. 

* Similar levels may be Identified with the Huean Operator, with High Level cognitive and 
planning functions at the top of the hierarchy, motor skill coordination at the intermediate 
level and autcultr and sensory systems at the lowest level. 

Coordination between the autonomous levels of the hierarchy ahd the Human Operator ia 
provided by three methods. At the lowest level coordination la enabled by providing the 
Operator with the ability to directly Interface with the robot arm and aeneore via force 
reflecting Joysticks and stereo camera systems. At the top of the hierarchy the coordination 
Is provided via interactive task planning activity between the Task Planner and the Operator. 
At the intermediate level the coordination ia achieved via a mechanism known am Traded/Shared 
control described further below. 


Traded control refers to the situation where the Operator relinquishes control ower process 
level operations for certain aegeente of the task sequence execution while retaining it for 
others. The transition between the operator controlled segments and autonomous program 
controlled segments le achieved via a 'hand-off* protocol that ensures that information about 
the vorld state is correctly communicated between the operator and the machine. 
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Shared control refers to the situation where the Operator and the autonosous progress 
Jointly participate In decision making at the process level. The contribution of the 
autonosous programs could be passive In nature ms In the form of monitoring operator actions, 
e. g. , ensuring that Joystick actions will not lead to possible collision with the 
environment. The participation may also be much more active with the autonomous programs 
actually contributing to the - manipulation of objects In the environment, e. g. • letting the 
operator control the wrist of the robot arm while the autonomous programs control Its global 
positioning. 

The overall structure of such a system architecture Is depicted In Figure 1. 

3. Run-Time Control Architecture Requirements and Constraints 

Ve now focus on the architecture for the Run-Time controller. The functions of this 
architecture are to: 

1) Mechanize Process Level Operations 

2) Permit Han-Hachlne Coordination 

The specific design of this architecture Is influenced by the context within which the 
telerobot operates, the details of which are described below x 

1 ) Han-ln-Loop 

This feature has the most significant Impact on the design of an RTC architecture. The 
existence of the capability of teleoperation Implies that the autonomous capabilities of the 
autonomous RTC architecture need not be complete. It is sufficient to have an architecture 
that is capable of performing autonomously for a reasonably high percentage of the times the 
task Is required. Since most computational algorithms to plan and mechanize simple physical 
subtasks such as grasps etc. are exponentially hard to perform, this feature of the 
architecture enables algorithms to be selected based on their likelihood of successful 
performance. 

The architecture Is also required to functionally Interface to both the Operator and the 
AIP planner. This Implies that an Interpreter style command language is required to enable 
the human to effectively comeunicate with the system. The Operator must also be able to 
interact with the system in a manner that does not Involve a knowledge of the detailed 
implementation and computational aspects of the system. This requirement is usually taken to 
mean the Implementation of a Task Oriented Language where actions are implicitly specified in 
terms of the required task. This concept is Implemented in the RTC via a Task Command 
Language augmented by a constraint specification scheme. The constraint specification 
language not only allows the operator (or the AIP) to designate the required physical task, 
but also allows the specification of a set of constraints that are to be naintalned by the 
autonomous system during the execution of such a task. The autonomous system then determines 
the appropriate actions to execute the task and also satisfy the constraints. A sample entry 
of the RTC command dictionary is shown in Figure 2. The operator is also required to 
interact with the system in a shared mode of operation as described earlier. The constraints 
specification command language allows a natural and easy extension to permit the 
specification of shared control activity. 

2) Specialized Environment 

The telerobot is required to operate in a space environment performing taskm much a • 
satellite repair, structure assembly etc. The specialized nature of the tasks implies that 
standard factory automation paradigms are inapplicable. Further, unlike a highly changing 
manufacturing environment, the space application environment does have a substantial body of 
off-line knowledge available in the form of actual astronaut experience, earth simulators 
etc. The existence of this knowledge indicates that script based techniques are feasible for 
decision making, because they permit the transfer of the available knowledge to the systea 
with the least effort. Of course, as the telerobot takes over more autonomous functions, the 
scripts could be replaced with modules capable of reasoning and generating their own 
activity. 

The other constraint of the specialized environment Is the perspective shaped by the 
fact that the Run-Time Controller will eventually fora part of an Embedded systea Implemented 
in space quallflable hardware/sof tware. Also, since the controller is part of an end-to-end 
telerobot system, requirements of overall system design affect the design of the local 
subsystem architecture. While these constraints are not rigid since the first 

implementations of the JPL telerobot are necessarily ground based research prototypes, the 
requirements for an evolutionary growth in the system require that these factors be 
considered while designing and building the baseline architecture. 

3) Parallel Activity Tracks 

A requirement of the telerobot architecture Is the management of parallel task execution 
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and the simultaneous use of aany physical resources available to the robot. Physical 
resources consist of various manipulation resources such as robot arms, end effectors, mobile 
platforms etc. , or various sensor resources. Each of these resource could be active 
simultaneously as In the example of dual arm Independent or coordinated actions. The 
architecture must be embedded with appropriate resource logic to ensure that resource 
conflicts do not occur vhen simultaneous activity tracks are In progress. Further, the error 
detection and recovery for each of these Independently controlled resources must be 
coordinated. 

4 > Computational Bottlenecks 

Computational resources must also be managed. Host robotics algorithms are 
computationally very Intensive, far outstripping the capabilities of uniprocessor machines. 
Concurrent processing In a distributed multiprocessor environment Is necessitated, and the 
system should therefore be capable of performing the dynamic computation task allocation 
necessary to optimally use the available computational resources available to the system. 

In the BTC, the computational paradigm that has been adopted Is the message passing 
multiprocessing system. 

5 ) Robust Operations 

Robust operations at the any level of the hierarchy require the laplementatlon of 
feedback, where the actual progress of a subtask execution In the world is monitored to guide 
further actions. At the sensor and servo level, this feedback Is Implemented via various 
control system schemes designed to operate with a continuous physical world. The RTC, on the 
other hand. Implements a feedback scheme that operates In event space. A more detailed 
description of an event space formulation of the RTC decision making Is given In the next 
section. 

4. Event Control 

The notion of event space and event control la Intimately connected with the time 
horlxon of the subsystem of Interest. It can be argued that reaction time for a subsystem at 
any level of the hierarchy should be sufficient to react to a possible world change that Is 
relevant to the goal commanded to that level. Therefore for any subsystem, its time horlxon 
should be comparable with the time constant of the control processes at that level. While the 
precise determination of appropriate time horizons must necessarily be ad-hoc. It 
nevertheless allows the definition of a hierarchical control scheme, where a higher level 
relies on the lover subsystem for the execution of commands associated with small tlae 
constants. 

A hierarchy in the chain of coaaand Is naturally coupled with a hierarchy In the 
processing of sensory information as well Cl 3. This leads to an Important slaplif ication In 
the overall task execution monitoring activity in the telerobot. That Is, the subsystem at 
the commanding level is relieved froa the mundane supervision of command execution at the 
lover levels, and is instead free to focus on monitoring and anticipating a smaller (and 
finite) number of crucial events. 

Taken together, a "trusted soldier principle" (21 nay be formulated. According to this 
principle the overall process-level decision space can be factored Into two distinct 
components. One component la acted upon by the RTC, while the other la the sole 
responsibility of the sensor and servo subsystems. 


On any level of hierarchy. Event Control requires the napping from the space of sensor 
event sequences: 

{Sj*}. i - 1. .M k K-l. .S n 

to the space of actuation event sequences available for this level: 

'{A i k >. i - 1 ..N r K-l. ,S n 

where *1* is the running index for the sequences and S n represents the number of lover level 
actuation/sensor subsystems. 

While this formalism can be used to describe a traditional servo system with digital 
elements in it. Its main goal Is to create a finite parameterization of actuation and sensor 
events therefore making error recovery through search possible. 

The "trusted soldier* principle leads to a unified command forsat. Commands to a lower 
level specify goals, constraints on the allowable ways to reach this goal, specifications on 
when activity should be stopped, and report formats as well. 
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In our caw th» coaaunlcatlon batvwn tht lavtla of th* want control apact < RTC > and the 
contlnuua apaca (HCH and SAP) la performed by a Primitive which la modelled as follows: 

Prlaltlva T - (P.H.F.R) 

vhara $ 

P * desired trajectory (position x forca apaca) 

H - desired claaa of control law 
F - daalrad raflax pradlcata (atop function) 

R - daalrad avant raportlng function 

Tha coaponanta P and H apaclfy tha phyalcal notion to ba axacutad by tha robot. The 
coaponant F dataralnaa tha taralnatlon of tha apaclflad phyalcal action wla a aanaor 
pradlcata which whan trua triggers tha approprlata raflax action (typically a atop). Tba 
coaponant R dataralnaa tha aanaor pradlcataa that ganarata tha avant raporta to tha RTC. 

Tha apaclf Icatlon of P and H dataralnaa tha daalrad aap froa tha actuation avant apaca to 
tha contlnuua aanaor /actuation apaca. Tha coaponant R apaclflaa tha aap froa tha aanaor 
apaca to tha aanaor avant apaca. Tha coaponant F apaclflaa tha pracoaputad raflax actlaaa 
that auat ba takan by tha HCH and rapraaant tha algabralc (aaaorylaaa) and local coaponanta 
of tha daelalon aap that ara to ba axacutad within tha fast daclalon tlaa conatanta of tha 
HCH aubayataa. We want to emphasize that tha atop function F la to ba daflnad bafora tha 
atart of tha actual coaaand execution. tharafora allowing praplanning for all poaalbla 
outcoaaa of T. 

Evant Control in tha RTC 

Tha lntarfaca batwaan tha RTC and tha lovar laval aubayataa adopts tha "trusted soldier* 
princlpla of oparation. That is. tha lovar laval aubayataa haa tha full raaponalblllty of 
parforalng tha daalrad coaaand T. If T is unsuccaaaful than tha rola of tha RTC la solwly 
that of coordinating and lntarpratlng tha various raporta R (froa all of tha aubayataaa) to 
dataralna tha naxt coaaand T to ba aant to ona of tha lovar laval aubayataaa. 

Glvan this aoda of oparation. conaldar tha actions to ba takan by tha RTC on racaptloa 
of a coaaand froa tha AIP or tha oparator. Tha coaaand aay ba avantually tranalatad according 
to ona of tha four following caaaa: 


1. COMMAND 

2. COMMAND 

3. COMMAND 

4. COMMAND 


~ > (P.M.F.R) 

— > (Pl.Mi.Fx.Ri) 

— ) ( p n*M n .F n .R n ) 

[ (Px .Mx . Fx . Rx) ( P2 * M2 * * * * ( Pflj'Mm . 1 


[(Pll.Mii.Fii.Rii) (Pi2*Mi2.Fi2' R 12) ( p lm'Mi 0 . p im' R lm) 1 

t ( p nl.M n i,F n i ,R n i) (Pn2' M n2 * p n2 * R n2) ••••( P nm • Mnm * p nni* R nnil 1 


Case 1 corraaponda to a slngla poaalbla prlaltlva T (albalt paraaatarlzad ) that can ba aant 
to tha lovar laval aubayataa. An axaapla can ba a coaaand to aova an ara to a glvan joint 
position using glvan joint intarpolatlon. 

Casa 2 corraaponda to tha caaa vhara ona of aany prlaltlvaa T aay ba aant to tha lovar 
laval aubayataa. A coaaand that apaclflaa and position, but not tha apaciflc trajactory to ba 
salactad by tha RTC. can ba uaad as an axaapla hara. 

Caaa 3 corraaponda to tha caaa vhara a aaquanca of prlaltlvaa T la nacaaaary for process- 
laval coaaand axacutlon. and caaa 4 daala with tha situation vhara aora than ona aaquanca la 
avallabla. 

Since only ona coasand can ba aand to a lovar laval at a tlaa. tha aablgulty that exists 
in tha cases 2 and 4 auat ba reaoved. Thera are aavaral poaalbla ways to do It. A "hard 
rule" approach assuaes an axlstanca of an lapllclt agraaaant between tha coaaandlng aubayataa 
and tha RTC on how a single fourplet should ba chosen. For axaapla It aay ba aasuaad that a 
straight line Intarpolatlon should ba always used. In this caaa there ara no real 
differences between cases 1 and 2. On the other hand a "soft rule* would allow tha RTC to 
rank all candidates and start execution fros tha "beat" ona. If it failed, and tha actios 
ware reversible. then another candidate could ba triad. Tha RTC lapleaenta a aixad approach. 
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It ranks possibilities internally based on an RTC internal criteria; but after a choice la 
aade and execution started* the rule beeoees "hard* • and an error in execution would require 
the RTC to report it back to the conserve! lng level without exploiting the rest of the options. 


Coaaanda to the RTC which result in Cases 1 and 2 are called Actions. Cossands 
resulting in Cases 3 and 4 are called Skills. Skills require a sequence of event control 
decision making prior to the Issuance of each lover level subsystes primitive T. In Case 3 
the sequence of primitives is constrained by the one possible mapping indicated earlier, but 
the numerical parameterization of the individual T is determined at run-time by the RTC. On 
the other hand* Case 4 allows the possibility that a different sequence of primitives 
together with their individual parameterization may be selected during the middle of the 
execution of a specific sequence. Since the creation and execution of this sequence is 
dynamically determined by the RTC, the ability to change the parameterization of a primitive 
T, or the actual change of a sequence, gives the RTC the ability to perform "trimming" <l.e. , 
error recovery of subtask command execution). Rote that by virtue of this definition of 
trimming* Actions cannot be trimmed. 

Using an analogy from control theory. the modules that Implement the * forward loop' 
operations of Action selection or a nominal Skill sequence determination are Incorporated 
into a Process Level Planner, and the * feedback loop' that analyzes events and determines the 
need for trimming is Incorporated in Ronltor, Predictor* Evaluator and Trlaaer modules. 
The feedback paradigm corresponding to this event control system is shown in Figure 3. 

It should be noted that two key assumptions govern the functionality and design of these 
modules. It is assumed that a nominal task sequence is known from off line analysis, and 
that in a case of ultimate failure, control can always be surrendered to the Operator. 

For the HCH a particular implementation of command components P, H and R is given by: 

P - a set of via points in Joint/task space, or spline functions. 

H - position mode or compliant aode control. 

R - standard report information returned to the RTC after stop (or reflex actions): 

joint positions* force/torque sensors readings, and the reason for end of execution. 

The choice of the stop functions set F is more complicated. It is clear that F must be 
complete in the sense that some condition must be eventually triggered. The triggering 
conditions aust also be unambiguous, namely only one stop function should be triggered at a 
time. Such a choice of the stop functions makes the number of possible outcomes for each HCH 
command finite. 

The process level planning la based on scripts. This allows the embedding of careful off- 
line analysis of the possible outcomes. In the future an expert system can be implemented to 
make a choice of an appropriate trimming action. Another advantage of scrlpta lies in their 
ability to Incorporate ad-hoc domain specific knowledge for planning of a nominal, feed- 
forward part of a skill. For example an approach position for a grasp can always be 
determined as a set distance away from the grasping point. Another specific feature of the 
current implementation is the existence of a command-object cross table which provides for 
every action, a list of parameters needed for nominal planning of this Action for each known 
object. It can be filled off-line or, if appropriate nodules are available* on-line upon 
demand. The nodular nat ire of this architecture allows system modification and extension in 
an orderly fashion. 

Skill Implementation includes two different phases: planning and execution. First an 

appropriate script employs a generate* test and modify paradigm to make a nominal sequence of 
actions. At the same time the script provides a list of possible trimming actions that can be 
employed to recover from the errors on the intermediate steps. Then the execution is 
performed according to the following loop: 

WHILE Actlon_Staek Not_Empty 
loop: 

Bind Action Parameters; 

Set_up Evaluatlon_context; 

Send Action to the HCH; 

Wait for Report; 

Evaluate Report; 

if SUCCESS then 
null ; 

else 

Determine trimming 

if ANY AVAILABLE then 

put trla_action on Actlon_Stack; 

else 

Report _up< Fail ) ; 
endlf ; 

endlf ; 

endloop; 
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An actual example of a acrlpt to parfora object grasping la ahovn In Figure 4. Numbers 
aftar aach action rafar to tha trlaaing stack associated with tha graap acrlpt. 

3. Iaplaaantatlon 

Tha previous aactlona have daacrlbad aoa a of tha techniques uaad to parfora cartala 
daalrabla alapllf lcatlona. auch aa aapplng a contlnuoua problaa apaca Into a discrete oaa. 
Nov tha atructura and iaplaaantatlon of tha RTC architecture will ba examined. 

Aa background, tha aoftvara Iaplaaantatlon of tha RTC la on an Al VAXatatlon II, uadar 
NlcroVHS. Tha aoftvara la vrlttan prlaarlly In Ada, vlth aoaa port Iona In C and Fortran, 
vhlch ara accaaaad vlth tha Intarfaca pragaa auppllad b/ Ada. Ada vaa our cholca of langaaga 
for tha RTC dallvarabla aoftvara, but va parfora rapid alaulatlon and prototyping of 
algorlthaa in Prolog, Llap, Saalltalk and a locally-vrittan davalopaant anvironaant callad 
Thraad. Our cholca of Ada for tha deliverables vaa aada for aavaral raaaona, tha anat 
obvloua balng that tha RTC vlll avantually hava to coaply vlth tha Spaca Station all-Ada 
aoftvara requirement anyvay. In addition, tha ability to axpraaa concurrent prograaa 
dlractly in tha languaga vlthout raaortlng to oparatlng ayataa lntarfacae vaa alao a 
daalrabla faatura. Our atrongasr reason, hovavar, vaa tha axtraaaly high nodularity of Ada, 
allowing aavaral people to vork on tha a ana aoftvara vlthout dlaaatroua conaaquencea. Thla, 
coupled vlth tha anoraoualy high level of coaplle-tlae error checking In Ada, turns out to 
hava bean a true expectation. Our aoftvara production haa bean only looeely coordinated 
aeong four different people, but It haa axpar lanced no inconpatlblllty problena and haa a 
hiatory of phanosenally lov levela of runtlsa errors. We believe that Ada la a good cholca 
for our dellverablen alnca they ara of precieely tha category of aoftvara Ada vaa daalgnad to 
fit, nanaly dlatributed eabedded real-time ayatena. Wa would make tha consent, hovavar, that 
Ada la not a good cholca of language to prototype In. 

Tha telerobot vlll ba uaad over the couraa of aavaral yeara for aany purpoaes. Including aa 
a testbed m vhlch to lnveatlgate both lov <e. g, feedback control lava) and aedlun (a. g. • 
graap atrateglaa) level robotica algorlthaa, aa vail as for tha targeted danonatration 
scenarios. For thla reason, tha architecture of tha RTC nust hava tha character sore of a 
davalopaant anvironaant rather than of a finished product. Tha ability to reconfigure the 
talerobot to parfora servicing tasks on satellites other than tha one targeted for tasting la 
one of tha obvious requlreaents. It Is our opinion, hovavar, that tha aany eoaplex 
interactions vhlch can taka place during various types of robot operations, even In auch a 
structured situation aa tha servicing of a modular satellite, vould aaka it likely that even 
state-of-the-art robotics algorithss vould ba far too rigid to seat all naada. To Insure 
sufficient flexibility, one must not only provide aa auch aa possible in the vay of current 
robotica algorlthaa, but also provide the ability to completely change hov, when and which 
algorithms are applied. 

Tha ability to do such general restructuring vithln a practical robotics lspleaentatlon has 
very large software consequences since the addition of a particular algorithm to tha alx uaad 
In tha robot might require brand new data representations to accommodate new Information 
relevant to previously unaodeled properties of objects, and the addition of the new algorithm 
could very veil require completely redesigning the strategies used for applying various 
currently-used algorithms. An example might ba tha addition of a new grasp planner module, 
vhlch might require that more data concerning each object to be grasped be installed in the 
data base and that the previously used strategy for stable grasp position determination be 
disabled or modified. 

In order to support such a high degree of flexibility, tha structure of tha RTC 
architecture is that of a framework in vhlch robotics algorithms can be installed, rather 
than a particular six of algorlthaa. The RTC conslats of several modules vlth a very rough 
degree of functionality assigned to each type of module, but the specific Inputs, outputs and 
details of operation which each module performs are controlled by reconf igurable data sets 
supplied to each module. The rough partitioning mentioned above is relatively simple and is 
an ad-hoc solution to the question of hov to slice up tha problem of processing tha commands 
received by the RTC. One important characteristic, namely the ability for the RTC to accept 
commands m parallel, had a major role in selecting tha problem meparatlon. 

The RTC is required to be able to perform more than one coaaand simultaneously, assuming 
that they do not conflict vlth each other. The task of managing access to various shsrable. 
non-shareable and queue-based resources is fairly straightforward in the context of tha 
talerobot, due to the fact that the autonomous system say ba Interrupted by the operator at 
any time. This makes it virtually Impossible to perform any time-scheduling of resources, so 
the simple restriction is made that all commands must ba independent of each other with 
respect to time shifts. This removes tha necessity of performing the many eoaplex sorts of 
resource allocation and activity coordination vhlch vould otherwise ba required between 
commands. This restriction is not as crippling as it seams, however, since there is no rule 
vhlch states that a single command can use only one resource. In fact, if a complex 
cooperative task involving two arms and a vision system is desired, it would simply need to 
ba formulated as a single coaaand so that the coordination operations necessary could take 
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pi ac* vi thin th* fraiatv ork of on* coamnd, without requiring propagation to other activities. 
This independence also greatly ■lspliflM the task of managing rfp oo— to w&nptcttd ar r or 
condition*, nine* all that la nscaaaary 1* a alapia tarml nation of tha command, or of all 
currently executing comm a nd* if tha arror la a global on*. Again, thla alapia bah a vi or la 
not raatrlctlva, alnca any aort of coaplax arror rmaponam can ba produced, if daalrad, a Imply 
by including it aa an axpectad poaalbla off-nominal altuatlon in a com ma nd. Obviously, thara 
la a graat daal of poaalbla interaction between robot action* in the real world, and 
formulating two coamanda in much a manner aa to be truly time-shift invariant might be very 
difficult in any given altuatlon, but thla la not really an la a u a . If the two operation* are 
unconnected, they can alaply be done in mar lam, formulated aa two a a p m rata co ama nd a. If they 
mu at be don* cooperatively, they ran be foraulatad aa one command. The performance of 
dependant operation* in parallel could aid efficiency, but la not a key requirement. 

kith tha above preface, the ad-hoc decompoei 1 1 of the RTC*s procaaalng of coamanda 
follow*. 

Co amend Farming 

• An Incoming command from the higher-level ayatem la flrat converted into m o me 
internal working data structure wear! by the other module*} thla allow* decoupling 
internal operation* from external interface*. 

Script elaboration 

• The command la then examined, and a apeciflc aequence of activitiea la decided upon 
to execute it. together with apeciflc poaalbllltlea for reaponaea to off-nominal 
event* during execution. An example of thla would be a graap command, for which a 
aequence of three atralght-lln* motion* followed by a closure of the gripper was 
decided upon, with the off-nominal behavior of backing up to the atartlng point and 
alaply repeating the entire motion if the graap doea not aucceed on the flrat try. 

Thla aequence la apeciflc in the number and type of motion primitive* to be executed, 
but no detail* are preaent yet aa to which trajectory la to be uaed or what the point 
of graap on the object la to be. 

Action Binding 

• An elemental motlon/aenalng primitive in the aequence 1* then further proceeded, 
in order to determine the precis* numerical parameter* for it* execution. Thla 
primitive la then aent to the appropriate subsystem. which reaponda with on* or more 
report* back to the RTC providing progreaa/reauit information about the command '* 
execution. 

Report Anal/ais 

• The returned report (a) are exaained. and a determination la made whether to 
continue execution, abort or take aome off-nominal corrective action, much aa the 
retry option mentioned above. If the declalon la made to continue, the prevloua step 
la then repeated for the next primitive in the aequence, followed by thi* one, until 
all primitives In the aequence have been executed. 

The overall architecture of the RTC le one In which each of the steps outlined above has 
been assigned to one type of module, with each command thus being processed by one of these 
four types of modules at any one time. Multiple modules execute In parallel and communicate 
by message passing In a very simple way. New instances of each module can be crested as 
needed to process parallel commands since each module le simply a worker which la dispatched 
with a job to do and then returns with a result, with no psrasnsnt memory of Its own mmd 
(with m single exception) no aids-sflscts. All information relevant to the processing and 
execution of the command is contained In the Inputs and outputs to each module. with the 
exception of spatial/geoaetrlc Information about the state of the world, which la contained 
In a global database. This database can be read by any module, but only the module assigned 
to analyze the reporta returned from an executing mubayatea haa the authority to write to it, 
thua updating the information m the database with the data returned by the executing 
subsystem. Thla is the mingle aide-effect present in the execution of any module, which 
removes the possibility of write contention and other such coordination problems. 

Thy RTC le thus mads up of zero or more copies of smeh of the four command -process lag 
modules, together with several additional module* which perform various essential function*: 

• An Interface Server, which serves aa a communication port to the other subsystem* 

In the telerobot. 

• A Decision Unit, which le the central dlepatcher/coordlnator for the RTC. Thla le 
simply a finite state machine, with the four-phase command processing behavior built 
into it. along with the terminate coammnd-on-error and resource- management behaviors. 

In the nominal came. It alaply feed* the Input command Into successive modules, 
propagating one module's output to the next module's input, with minor exceptions. 

In anomalous situations, it alaply takes several straight! orvard steps to insure that 
the executing subsystems and Internal module* working on that command are shut down 
and sends a report of the halt back to the higher levml system commanding the RTC. 
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kcauM of th# fllapllclty of th* Dvclaion Unit Cn alaplt daclaloo tm) v It ntada 
only a vary short parlod of tiaa to asacvta any >?a pona> to on avmt f such aa 
an iBcoalng aaaa agt. For most c a a aa . It alaply atnda out tha locoalog data to t ho 
app ropr lato module. Thla allows wary rapid ra sp o n s a to ex ce ption conditions* such as 
a command from tha telero b ot *s human oparator to stop executing. Tha Daclalon Unit 
will almost cartalnly ba ldla within a law mil lisa c o nd s of receiving tha mammmga. and 
can than pro cass It* and sand out aubsyatam halt prlaltlwas wary rapidly* giving only 

• faw tans of nllllaaconda of daisy batwaan tha oparator * a commanding tha halt and 
tha arm actuation aubsyatam halting arm motion. 

• Command Parsar modulus* aa many as naadad* which conwart tha Input to aa Internal 
working form. It should ba explained that one desirable feature of tha talarobot Is 
to let the oparator Intervene In execution at any level of tha command hierarchy from 
A I ( activity planning Interface) down to detailed are motions (teleoperstlool* so tha 
RTC'e co m m a nd Inputs are also of human -usable fora. Tha c om m and Input from tha 
higher level is thus In the fora of an ascii string with fairly readable content. 
From the RTC viewpoint than, there Is no distinction made between the human operator 
and tha AX planner. 

• Script Elaborator modules* as many aa needed* which decide on the basic script to 
follow for executing the command. In cases where a parameter for aa earlier 
primitive action must be determined by using a precise value for a parameter to an 
action to ba executed later* <e.g.* a precise grasp point may ba naadad to backtrack 
a trajectory to the current position of the arm) then determination of these 
numerical parameters la also performed at this phase* and the -la. ailed parameters are 
Inserted Into the primitive before It la aent to the next phase. This permits the 
use of backtracking aa a planning technique If desired* along with any other 
technique which requires a noncauaal determination of specific parameters for 
actions. 

• Action Binder modules* aa many aa needed* which determine the precise numerical 
values needed for each aubaystea primitive to be sent out for execution. The Action 
Binder would ba Invoked once for each primitive In the script* whether tha primitive 
was a nominal action or a raaponma to soae planned -for off-nominal condition. The 
stepwise Invocation of the Action Binder allows the use of parameters determined at 
one point of the execution of the command during run-time to be used at later points 
In a very simple fashion. Also* as described above* a parameter for an action 
primitive may be already fixed when It arrives In an Action Binder* which will not 
disrupt the normal operation of the module. 

• Analysis Unit modules* as many as needed* which examine the reports returned by the 
executing subsystems* update the global geometric/epatial database accordingly* and 
determine the recommended course of action to be followed* which would be to continue 
normally, abort* or take a specific foreseen corrective action. This recommendation 
is then sent to the Decision Unit* which say choose to follow It* If overall 
execution Is normal* or say Ignore It and choose to terminate the command* If for 
example the AI has sent a halt Instruction to the RTC during the execution of the 
command. The key point concerning the AU modules Is that they operate entirely on 
command-related Information* and do not take Into account such things as global 
anomalies* but operate as a memoryless single-input single-output system. The Input 
is the command context together with the report sent back by the subsystem* and the 
output is the recommendation for action. The Decision Unit worries about 
coordinating any overall behavior which crosses command boundaries. 

• A Database Server module* which simply behaves as a shared-memory area for the 
other modules. Any given location In It can be written to by only one AU at any 
given time* and there are various validity flags to Indicate whether or not other 
sodules should be allowed to read any given piece of Information. 

• In addition to these sodules* which form the basic configuration of the RTC as 
shown in Figure S* there are several others which will not be detailed* but which 
exist because of the requirement that the RTC's geometric database be a ccess ible to 
any subsystem in the telerobot* to act as a central repository. Also* there Is the 
requirement that the AI plmnnar * in order to perform its planning* say need to use 
backtracking methods. To support this* the RTC contains an Identical copy of the 
Command Farmer /Script Elaborator/ Action Binder nodules which are used as a 
hypothesis-testing facility by the AI planner. The planner can simulate as such as 
possible of the execution of s possible RTC command* without actually commanding any 
subsystem activity* so as to determine whether or not a particular line of planning 
will be found to be Infeasible at the numeric level by the RTC. This facility allows 
the fairly ad-hoc division In the planning made between the AI planner and the RTC to 
function robustly In the presence of couplings between the symbolic and numeric 
levels of robot activities. 
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Ths pri mary aaptct of the oodulto which perform command procoMlng la that thty ara 
Implamantad not aa hardwired entitle* which operate on data, but aa data- drive* 
* in te r pr e t er * , * which take aa input a program, aa well aa data. Thle impl em e n tation aXlowa 
the module* to be eaaily reconfigured to uee a new algorithm if at all poaalble. For example, 
adding a new algorithm may abeolutely require a complete reetructurlng of the existing a y at em 
• be ca me* of robotic* conml deration*, but much reetructurlng would not be neceemary elmply 
be cau e e of mof tware dlfflcultie*. Thle feature allow* the addition of many type* of robotic* 
and aleo II technique* to the FTC if deaired einee there ia no reetrlction at all on what 
data la passed between the module*, and only trivial reetrlction* on the order in which they 
are invoked. It la our hope that later veralona of the Script Elaborator will uee Al-type 
reeeoolng technique* to produce the baalc acrlpta for uee during emecutloo, rather than the 
cut -and -paste/ table lookup technique ueed now. Thle vould give the ability to literally 
replan a aeq uence in the event of an error, instead of forcing error r aa p o na aa into a pa c ific 
script*. Aleo, a prediction capability, capable of utilizing expectation information la 
analyzing the tensor data during execution, vould be a desirable feature to add to the 
Analysis Unit. The ability to re mo w the Independency restriction on separate c omma n ds and 
perform coordination between more than one command vould alto be useful in in c r e asing syst em 
efficiency. All of these ldeee haw been explored in a preliminary fashioa, and several 
fairly stralghtforvard implementation alternatives have been found to e e ch of thee, 
indicating that the ability to specify module behavior at run time, rather than Jutt its 
input data, is an extremely powerful feature of the RTC. 

A feature of the RTC is the feet that the memo ryleea, dispatched -worker module format 1* 
ideal for Implementation on a multiprocessor hardware architecture. Modules do not perform 
any significant nonlocal reference* (the exception 1m database access, which can be reduced 
by elmply grouping read requests into bunches, rather than lot* of individual read request*} 
and do not require any communication among themsmlvea during execution. A preliminary study 
indicates that conversion of the RTC to operate on a hypercube multiprocessor vtwld be e very 
straightforward task. 

Another aspect of the RTC'* Internal operation la that it la relatively simple to treat the 
human operator am a subsystem to be commanded by the RTC. This allow several simple but 
effective solutions to the extremely difficult problems of specifying to the autonomous 
systes what the intentions and outcomes of human teleoperation activity are. For example, if 
the operator intervenes Into autonomous execution and picks up an object, there is no vay for 
the autonomous systes to examine are trajectories end gripper force date to determine that 
the object warn grasped at all, or if it warn grasped vhat the position of it in the gripper 
lm. This very simple example show hov difficult is the task of sharing control of as 
autonomous systes with a human operator. One solution, which by necessity imposes a good 
deal of overhead and restriction on the human operator, is to specify intervention activities 
to the telerobot In the same form mm any other command, with the exception that the 
performing subsystem la the operator. This paradigm vould mean that precise numerical 

parameters vould be left out of each step, but the normal aequence of subsystem primitives 
vould be used by the RTC. and likewise the usual sequence of RTC cosmsnds vould be sent out 
by the AI, and the operator vould face the restriction cf performing only that portion of the 
task specified by the primitive. An exploration of this method of structured operator 
Intervention, which Is one candidate for the JPL. telerobot. Is given below. 

If an object could not be reliably grmmp^d by the autonomous system, the operator could 
Instruct the AI planner to 

•grasp Object with Right_arm via Oper*tor_intervention. * 

This vould result in the command trickling down the hierarchy through tbe RTC and m 
subsystem primitive to the operator appearing on the control console. In this example, tbe 
first might be: 

"move Rlght.ara to_neighborhood_of Object*. 

The operator vould perform this. In teleoperation mode, and Indicate that he vaa finished. 
The RTC vould then send out the next primitive: 

•move Right_ara to_gra*p_point_of Object*. 

The operator vould comply, seating the gripper on the object In a satisfactory 
configuration so that when the gripper vam closed. It vould grasp the object firmly, without 
movlno the object. The RTC vould then be able, by performing kinematics and geommtrlc 
computations, to know vhat the grasp point reference frase for the object was and vould. 
therefore, be able to correctly update the data base as to the position of tbe object after 
it has been grasped. The final priaitive vould be sent by the RTC, directing tbe operator to 
close the gripper, and the operator *m response vould confirm that the grasp took place as 
expected, without disturbing tbe spatial relationship mat up. 

This scsnarlo demonstrates the additional overhead Imposed on the operator by the necessity 
of aalntalnlng the autonomous ayatea'a Integrity. It la esssntlal, however, that some form 
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of strong rMtriction b« placed on the operator, . not only to coordinate activity with thw 
autoauMoue syatea, but to p reve nt operator disruption of the autonoaoue eystee duo to buaaa 
oversight. Xt Is likely that there will need to be sore effort put Into protecting tbs 
autonooous sys ten Sr os the operator than Into salting the aotonoaous sy s tem operate 
effectively on Its own. 

Another Issue we bellewe to be of foremost Importance In the telerobot design Is that of 
detection of anomalous conditions In the world. Even In such a highly Identified environme n t 
as satellite servicing. It Is crucial that sensor feedback be employed as often as possible 
so as to prevent any cascade a if errors forward through the execution of e servicing task. It 
Is unlikely that any AX /robotics autonomous system will be able to make allowances for error 
propagation in its activities, within the near future. Such propagation must be eliminated 
If the autonomous syatea Is not to ba c o s e completely confused, with major portions of Its 
world model Invalid, vhlch Is s c ompletely unacceptable aituation owing to the large amount 
of time and effort vhlch vould likely be required In order to restore the world model to a 
co r re ct atate (work weeka). 

The development of such an architecture must nece ssa rily recognize the 11 *i tat lone of 
current science end technology In this nascent area. Early architectures focus oa 
Integrating the process-level autonomous functions into the syatea for simple, independently 
controlled axes. Operator -machine coordination is restricted to simple traded control 
schemes. Later architectures support sore complex physical environments (sore arms, redundant 
area), as veil as sore complicated f unctlonallty much as coordinated arm motion and true 
shared control . Concomitant with this incre a sed complexity end functionality is the 
management of complex computational architectures and the integration of sore sophisticated 
error recovery and planning methods into the systes. At the present tlse, the ETC of the JPL 
Telerobot hms been implemented to contain the following capabilltleet 

• Command Paraer — parses amcll strings fros a simple BNP form into m record data 
structure containing equivalent information. The BNF language in uae has roughly 1M 
terminal aymbola, SO clauaea. 

• Script Elaborator — uaes a almple decision tree to splice portions of eoaaand 
sequences together into a coherent script, together with off-noalnel scripts. Has 
the ability to backtrack from goal point for purposes of trajectory generation. 

• Action Binder — performs generation of jolnt/taak space linearly Interpolated 
trajectories in e plecevlse fashion, using kinematic and load-carrying constraints of 
the manipulator arss, together vith collision detection, to plan small arm motions 
for grasping purposes. 

• Analysis Unit — uses a table-driven decision tree to evaluate the outcoee of an 
execution and makes a re c om m endation. If the recommendation requires corrective 
activities, then the script to perform it Is aiaply looked up In a table, having been 
generated by the Script Elaborator. 


6. Conclusion 

We have deecrlbed a Run-Time Control architecture for the JPL Telerobot and dlscuaewd 
associated iaauea. This work vas performed at the California Institute of Technology, Jet 
Propulsion Laboratory under a contract froa the National Aeronautics and Space Ada Inst ration. 
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